Method of cyberthreat detection by learning first-order rules on large-scale social media

ABSTRACT

A cyberthreat detection method and system includes a distributed file system and a commodity cluster configured in data communication via a network, wherein the commodity cluster is defined as a plurality m of servers, each including a computer processor and a non-transitory computer-readable storage medium. The system and method includes receiving a data array characterized by a key and a value in a set of pairs relating to social media posts and users, storing a plurality of predetermined ground predicates, constructing a ground predicate graph for each user reflected in the array, constructing a user centric graph having one or more vertices and one or more edges and wherein each vertex represents the ground predicate graph corresponding to each user. The method includes partitioning the user centric graph into balanced portions P i  corresponding to the number of servers and wherein the ground predicates of each vertex in partition P i  are stored as a file on a server associated with that partition P i . The system and method determines a plurality of learned rules, in parallel on each server, on the files stored on each server. The system and method also includes receiving a union of the plurality of learned rules and determining a respective weight for each of the plurality of learned rules of the union. The system and method ranks the plurality of rules in order of accuracy by the plurality of weights.

GOVERNMENT INTEREST

The invention described herein may be manufactured and used by or forthe Government of the United States for all governmental purposeswithout the payment of any royalty.

FIELD OF THE INVENTION

The present invention relates to systems and methods for protectingagainst a cyberthreat on large scale social media. The field relates toan application of first order logic, probability theory, and graphtheory in implementing a method. In particular, the field relates tomethods to learn first-order rules on large-scale Twitter® data using acommodity cluster.

BACKGROUND

Twitter® is a microblogging service where users can instantaneouslycommunicate with others by publicly posting short text messages of up to140 or 280 characters, for example. Twitter® has been widely used forpolitical campaigns, marketing and advertising, sharing breaking news,and communication during catastrophic events like earthquakes andtsunamis.

Social media websites like Twitter® have been exploited by criminals andsaboteurs through cyberattacks against account users and theirorganizations. Hackers have taken control of government accounts andhave posted false information from popular accounts. The damageinflicted by hackers and wrongdoers is oftentimes exacerbated when falsepostings are reproduced on a large scale. Furthermore, malware may bespread through social media posts when an unsuspecting user is dupedinto clicking a malicious link presented thereon.

These threats are not only socially damaging, but economically damagingas well. Thus, protecting users and systems against cyberthreats onsocial media has become a critical aspect of enterprise securitymanagement.

However, there are technical challenges that arise in designing asuitable method or system that can model and reason about the veracityof social media posts. One challenge is to represent the complex anddiverse social media data in a principled manner. For example, a tweetis a 140-character message posted by users on Twitter®. It isrepresented using 100+ attributes, and attribute values can be missingand noisy. New attributes may appear in tweets; some attributes may notappear in a tweet. Hashtags, which begin with the # symbol, are usedfrequently by users in tweets to indicate specific topics or categories.There are thousands of hashtags in use today; the popularity of ahashtag changes over time. Some hashtags may become trending/popularduring a particular time period. Another challenge is to construct aknowledge base (KB) on social media posts. The goal is to learn theentities, facts, and rules from a large number of posts. Anotherchallenge is to reason about the veracity of the posts using the KBcontaining a large number of entities and facts. Thus, suspiciouscontent/activities can be flagged as soon as possible to discoveremerging cyber threats.

Ascertaining the veracity (or trustworthiness) of social media posts isbecoming very important today. For this, one must consider both thecontent as well as users' behavior. patent application Ser. No.15/585,397, filed May 3, 2017, which names the same inventive entity asthis application, the contents of which are hereby incorporated byreference in their entirety, disclosed a method to model and reasonabout the veracity of tweets to discover suspicious users and maliciouscontent on Twitter®. This earlier method used the concept of Markovlogic networks (MLNs) for knowledge representation and reasoning underuncertainty. It was developed to analyze both the behavior of users andthe nature of their posts to ultimately discover potential cyberattackson Twitter®. The earlier system embodies a knowledge base (KB) overtweets to capture both the behavior of users and the nature of theirposts. The KB is defined by a set of first-order logic rules/formulasalong with weights. Using probabilistic inference on the KB, the earliermethod identified malicious content and suspicious users on a givencollection of tweets. In this method, the rules in the KB werehandcrafted prior to system execution. However, malicious users onTwitter® may continue to evade being detected by changing the way inwhich they pose cyber threats on Twitter®. Thus, as the number of tweetsbeing analyzed in the KB becomes very large, interesting new rules mayappear and disappear and change over time. Consequently, a new solutionset forth in this application is directed to a system that automaticallylearns the structure of the KB, i.e., the first-order rules (alsoreferred to as formulas) so as to more accurately predict and identifymisinformation potentially indicative of cyberthreats.

SUMMARY OF THE INVENTION

With the above in mind, an embodiment of the present invention isrelated to a cyberthreat detection system comprising a distributed filesystem and a commodity cluster configured in data communication via anetwork, wherein the commodity cluster is defined as a plurality m ofservers, each including a computer processor and a non-transitorycomputer-readable storage medium comprising a plurality of instructionswhich, when executed by the computer processor, perform a method.

The method includes receiving a data array characterized by a key and avalue in a set of pairs relating to social media posts and users,storing a plurality of predetermined ground predicates, constructing aground predicate graph for each user reflected in the array,constructing a user centric graph having one or more vertices and one ormore edges and wherein each vertex represents the ground predicate graphcorresponding to each user. The method includes partitioning the usercentric graph into balanced portions P_(i) corresponding to the numberof servers and wherein the ground predicates of each vertex in partitionP_(i) are stored as a file on a server associated with that partitionP_(i), and determining a plurality of learned rules, in parallel on eachserver, on the files stored on each server. The method also includesreceiving a union of the plurality of learned rules and determining arespective weight for each of the plurality of learned rules of theunion. In addition, the processor ranks the plurality of rules of theunion by the plurality of weights.

The system may be further defined by variables in the predeterminedground predicates comprising one or more of the following types: (i)“tweetID” to denote the ID of a tweet (ii) “userID” to denote the ID ofa user (iii) “link” to denote a URL (iv) “hashtag” to denote a wordprefixed by the ‘#’ symbol and (v) “count” to denote a non-negativeinteger.

The system may be further defined by having the data array split intoseparate data blocks and wherein constructing the user centric graphcomprises performing a map operation in parallel on each block, andshuffling and sorting the data output from the map operations. Thesystem may also include a processor configured to perform a reduceoperation in parallel for each block of data shuffled and sorted, andstoring the data output from the reduce operation in a file.

The system may be further defined by having the file comprise a HadoopDistributed File System file. In addition, the system may be furtherinclude defining an edge defined between a first and a second user inthe user centric graph if any of the following conditions are satisfied:a tweet of the first user mentions the second user; a tweet of thesecond user mentions the first user; the first user is a friend of thesecond user; the second user is a friend of the first user; the firstuser is followed by the second user; the second user is followed by thefirst user; a tweet of the first user was retweeted by the second user;or a tweet of the second user was retweeted by the first user.

The system may further include ranking the plurality of rules of theunion by determining a respective partial product of the weights foreach of the plurality of learned rules of the union and summing theplurality of partial products to define a total weight.

Another embodiment of the invention may include a computer-implementedmethod of cyberthreat detection, whereby the method involves receiving,using each of a plurality m of servers, a plurality of first-orderpredicates. The method includes receiving, using a distributed filesystem, a plurality of ground predicates associated with the first-orderpredicates and each characterized by a key and a value. The methodincludes receiving a data array characterized by a key and a value in aset of pairs relating to social media posts and users, and storing theplurality of ground predicates. The method includes constructing aground predicate graph for each user reflected in the array, andconstructing a user centric graph having one or more vertices and one ormore edges and wherein each vertex represents the ground predicate graphcorresponding to each user. The method includes partitioning the usercentric graph into balanced portions Pi corresponding to the number ofservers and wherein the ground predicates of each vertex in partition Piare stored as a file on a server associated with that partition Pi. Themethod includes determining a plurality of learned rules, in parallel oneach server, on the files stored on each server, and receiving a unionof the plurality of learned rules, and determining a respective weightfor each of the plurality of learned rules of the union. The methodincludes ranking the plurality of rules of the union by the plurality ofweights.

The method may further include selecting the ground predicates to be oneor more of the following types: (i) “tweetID” to denote the ID of atweet (ii) “userID” to denote the ID of a user (iii) “link” to denote aURL (iv) “hashtag” to denote a word prefixed by the ‘#’ symbol and (v)“count” to denote a non-negative integer.

The method may include splitting the data array into separate datablocks and wherein constructing the user centric graph comprisesperforming a map operation in parallel on each block, and shuffling andsorting the data output from the map operations. The method may alsoinclude a performing a reduce operation in parallel for each block ofdata shuffled and sorted, and storing the data output from the reduceoperation in a file.

The method may be further defined by providing a file comprising aHadoop Distributed File System file. The method may further includesplitting the data into key-value pairs, performing a second mapoperation in parallel in order to output weights, shuffling and sortingthe output weights from the second map operation, performing a secondreduce operation in parallel on the output weights to produce the usercentric graph, and storing the user centric graph in the file.

The method may further include defining an edge between a first and asecond user in the user centric graph if any of the following conditionsare satisfied: a tweet of the first user mentions the second user; atweet of the second user mentions the first user; the first user is afriend of the second user; the second user is a friend of the firstuser; the first user is followed by the second user; the second user isfollowed by the first user; a tweet of the first user was retweeted bythe second user; or a tweet of the second user was retweeted by thefirst user.

The method may further include ranking the plurality of rules of theunion by determining a respective partial product of the weights foreach of the plurality of learned rules of the union and summing theplurality of partial products to define a total weight.

Another embodiment of the invention may include a system for detectingsuspicious activity on social media comprising a distributed file systemand a commodity cluster configured in data communication via a network,wherein the commodity cluster is defined as a plurality m of servers,each server includes a computer processor and a non-transitorycomputer-readable storage medium, a data array characterized by a keyand a value in a set of pairs relating to social media posts and users,and wherein the processor is configured to store a plurality ofpredetermined ground predicates. The system includes the processor beingconfigured to construct a ground predicate graph for each user reflectedin the array, and wherein the processor is configured to construct auser centric graph having one or more vertices and one or more edges andwherein each vertex represents the ground predicate graph correspondingto each user. The system includes the processor being configured topartition the user centric graph into balanced portions Pi correspondingto the number of servers and store the ground predicates of each vertexin partition Pi as a file on a server associated with that partition Pi,and wherein the processor is configured to determine a plurality oflearned rules, in parallel on each server, on the files stored on eachserver. The system includes a processor configured to perform a unionoperation to generate a union of the plurality of learned rules, andwherein the processor is configured to determine a respective weight foreach of the plurality of learned rules of the union. The system includesthe processor being configured to rank the plurality of rules of theunion by the plurality of weights.

The system may further include the processor being configured todetermine the plurality of learned rules by running a Markov logicnetwork learning algorithm.

The system may further include having an edge be defined between a firstand a second user in the user centric graph if any of the followingconditions are satisfied: a tweet of the first user mentions the seconduser; a tweet of the second user mentions the first user; the first useris a friend of the second user; the second user is a friend of the firstuser; the first user is followed by the second user; the second user isfollowed by the first user; a tweet of the first user was retweeted bythe second user; or a tweet of the second user was retweeted by thefirst user. The system may further the processor being configured torank the plurality of rules of the union by determining a respectivepartial product of the weights for each of the plurality of learnedrules of the union and summing the plurality of partial products todefine a total weight.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating an overview of the method ofcyberthreat detection on large scale social media according to anembodiment of the invention.

FIG. 2 is a data structure relationship diagram according to theinvention illustrated in FIG. 1.

FIG. 3 is a series of data structure relationship graphs and diagramsillustrating ground predicate mapping to a user-centric single vertexwith defined edges according to the invention illustrated in FIG. 1.

FIG. 4 is a flowchart illustrating ground predicates being grouped byuser ID in parallel according to the invention illustrated in FIG. 1.

FIGS. 5A and 5B are a flowchart illustrating mapping of groundpredicates by user ID according to the invention illustrated in FIG. 1.

FIG. 6 is a flowchart illustrating a reduction operation according tothe invention illustrated in FIG. 1.

FIG. 7 is a flowchart illustrating the construction of a user-centricgraph in parallel according to the invention illustrated in FIG. 1.

FIG. 8 is a flowchart illustrating a map operation performed duringconstruction of the user centric graph of FIG. 7 according to theinvention illustrated in FIG. 1.

FIG. 9 is a flowchart illustrating a reduction operation performedduring construction of the user centric graph of FIG. 7 according to theinvention illustrated in FIG. 1.

FIG. 10 is a flowchart illustrating user-centric graph partitioning andsubsequent use according to the invention illustrated in FIG. 1.

FIG. 11 is a flowchart illustrating a combination and sorting of learnedrule weights on different servers according to the invention illustratedin FIG. 1.

FIG. 12 is a schematic diagram of an exemplary computerized systemaccording to an embodiment of the present invention.

FIG. 13 is a schematic diagram of exemplary components of thecomputerized system of FIG. 12.

FIG. 14 illustrates examples of learned formulas resulting from step1009 in FIG. 10.

FIG. 15 illustrates examples of weights output in step 1012 in FIG. 10for the learned formulas resulting from step 1009 in FIG. 10.

DETAILED DESCRIPTION OF THE INVENTION

The present invention will now be described more fully hereinafter withreference to the accompanying drawings, in which preferred embodimentsof the invention are shown. This invention may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the invention to those skilled in the art. Those ofordinary skill in the art realize that the following descriptions of theembodiments of the present invention are illustrative and are notintended to be limiting in any way. Other embodiments of the presentinvention will readily suggest themselves to such skilled persons havingthe benefit of this disclosure. Like numbers refer to like elementsthroughout.

Although the following detailed description contains many specifics forthe purposes of illustration, anyone of ordinary skill in the art willappreciate that many variations and alterations to the following detailsare within the scope of the invention. Accordingly, the followingembodiments of the invention are set forth without any loss ofgenerality to, and without imposing limitations upon, the claimedinvention.

In this detailed description of the present invention, a person skilledin the art should note that directional terms, such as “above,” “below,”“upper,” “lower,” and other like terms are used for the convenience ofthe reader in reference to the drawings. Also, a person skilled in theart should notice this description may contain other terminology toconvey position, orientation, and direction without departing from theprinciples of the present invention.

Furthermore, in this detailed description, a person skilled in the artshould note that quantitative qualifying terms such as “generally,”“substantially,” “mostly,” and other terms are used, in general, to meanthat the referred to object, characteristic, or quality constitutes amajority of the subject of the reference. The meaning of any of theseterms is dependent upon the context within which it is used, and themeaning may be expressly modified. An embodiment of the invention, asshown and described by the various figures and accompanying text,provides for systems and methods of detecting and protecting against acyberthreat on large scale social media. In particular, a method tolearn first-order rules on large-scale Twitter® data using a commoditycluster. However, the embodiments of the invention described herein maybe relevant to other social media platforms. Nevertheless, the focus ofthe present invention is directed toward Twitter® and text posts of upto 140 or 280 characters from Twitter® users, defined as tweets.

As will be described in more detail below, one aspect of this solutioncomprises a system and method to learn relevant rules in the KB over alarge number of tweets using a divide-and-conquer approach in acommodity cluster with multiple servers. This system can be described asan application of first-order logic, probability theory and graph theoryprinciples. In graph theory, a graph comprises a mathematical structureused to model pairwise relations between objects. A graph in thiscontext is represented by vertices (or nodes) that are connected byedges (or lines).

In statistical relational learning, a Markov Logic Network (MLN) is analgorithm or representation that combines first-order logic andprobabilistic graphical models. Examples of MLNs are disclosed in M.Richardson and P. Domingos. Markov Logic Networks; Machine Learning,62(1-2):107-136, Jan. 27, 2006, the contents of which are herebyincorporated by reference in their entirety. Formally, a MLN is a KBdefined by a set of pairs (F,w), where F is a first-order formula (orrule) that denotes a constraint and w is a real-valued weight of theformula. A formula with a positive weight is more likely to be satisfiedin a possible world. Conversely, a formula with a negative weight ismore likely to be unsatisfied (in other words, less likely to besatisfied). The higher the magnitude of the weight, the stronger theformula is as a constraint and thus importance.

A grounding of a formula (or predicate) is obtained by replacing all itsvariables by constants from a data set. The obtained formula (orpredicate) is called a ground formula (or ground predicate). Based onthe predicates in the KB, tweets, and external data sources, groundpredicates are generated for further processing on this system.

The system described below constructs a user-centric graph on thegenerated ground predicates in parallel. This graph is a weighted graphwith weights on the vertices and edges. It essentially clusters groundpredicates pivoted around the social media users, as the goal of the KBis to reason about suspicious users and malicious content on Twitter®.By applying parallel graph partitioning, the user-centric graph ispartitioned to organize ground predicates into m partitions to reducethe chance of missing important rules that may span across multiplepartitions. Each partition can be processed by a single server in thecluster, and first-order rules are automatically learned on eachpartition using an existing MLN structure learning technique. The unionof the rules from all partitions is computed. In parallel, the weightsof all the rules are learned on each partition using an existing MLNweight learning technique. Finally, the method embodies a ranking schemethat objectively combines the learned weights and retains only the mostrelevant rules in the KB. The parallel graph partitioning and thearrangement of the server cluster achieve a greatly enhanced speed andflexibility for the learning system as compared to prior systems.

Because structured learning on a large number of ground predicatesgenerated from tweets (e.g., 1+ million ground predicates) can be veryslow, the current invention employs a divide-and-conquer approach toexpedite learning the most relevant rules of a knowledge base(simplified as “KB”). As the main purpose of the KB is to analyze datafrom Twitter® to detect suspicious users and malicious content, itfocuses on users in tweets so that ground predicates can be partitionedamong servers appropriately. As depicted, and will be explained infurther detail, the rules are learned in parallel with emphasis placedon reducing the chance of missing important rules that may span acrossseveral partitions/servers.

Referring to FIG. 1, a flowchart illustrating an overview of the methodfor protecting against a cyberthreat on social media, such as, forexample, Twitter®. As shown, the invention is designed to operate on acommodity cluster with multiple servers 101, with stored data using whatis known in the art as a Hadoop Distributed File System (HDFS) 102. Oneexample of such a system is disclosed by K. Shvachko, H. Kuang, S.Radia, and R. Chansler: “The Hadoop Distributed File System,” 2010 IEEE26th Symposium on Mass Storage Systems and Technologies (MSST), May2010, the contents of which are hereby incorporated by reference intheir entirety. The tweets 103 may be processed, and ground predicates104 may be stored in HDFS. By processing the ground predicates, theinvention may perform graph construction in parallel 105 and may outputa user-centric graph 106. Each vertex and associated edge of the graphhas a weight associated with it. The user-centric graph may bepartitioned into m balanced partitions/servers by reducing or minimizingthe total weight of the cut edges 107. Furthermore, ground predicatesassociated with each partition/server may be grouped 107 and then storedin m files 108.

Next, each file containing ground predicates may be processed by anMLN-structure-learning-software on one server 109. The method maycombine the rules learned on each partition/server by applying a unionoperation 110. After that, the method may learn the weight of each ruleon each partition/server using what is known in the art asMLN-weight-learning-software 111. Finally, the rules may be combinedobjectively and ranked 112 to output those rules that are the mostrelevant.

The divide-and-conquer approach described herein is facilitated by theuser-centric graph 106, which serves three purposes: First, a vertex ofthe graph enables the ground predicates to be grouped in the data basedfor a specific user; Second, an edge within the graph captures theextent of social relationships between two users based on groundpredicates that are true in the data; and Third, by assigning weights tovertices and edges of the graph, the invention can dynamically frame anissue and find an appropriate resolution. It does this in a way tocreate balanced partitions/servers of ground predicates for structurelearning while minimizing the chance of missing relevant rules acrosspartitions. As a result, rule learning time is reduced by operating inparallel on smaller sets of ground predicates rather than operating on asingle large set.

FIG. 2 is a data structure relationship diagram that illustrates theinvention's first-order predicates from the Twitter® KB. The vertices inthe graph denote the predicates. For example, the predicates used in thepresent embodiments are: tweeted(userID, tweetID) 201, mentions(tweetID,userID) 202, containsLink(tweetID, link) 203, malicious(link) 204,retweeted(userID, tweetID) 205, containsHashtag(tweetID, hashtag) 206,isPossiblySensitive(tweetID) 207, trending(hashtag) 208,retweetCount(tweetID, count) 209, verified(userID) 210, friend(userID1,userID2) 211, attacker(userID) 212, isFollowedBy(userID1,userID2) 213,friendsCount(userID, count) 214, followersCount(userID, count) 215,statusesCount(userID, count) 216, and favouritesCount(userID, count)217.

The variables in the predicates are of the following types (i) “tweetID”to denote the ID of a tweet (ii) “userID” to denote the ID of a user(iii) “link” to denote a URL (iv) “hashtag” to denote a word prefixed bythe ‘#’ symbol and (v) “count” to denote a non-negative integer.

An edge, shown by a curved line, located between two predicatesindicates that they can appear in a first-order rule. Given the densenature of the graph, it is evident that a large number of candidaterules may be tested to extract the most relevant rules from a set ofground predicates. A ground predicate may be obtained by replacing thevariables in a predicate by constants from the data.

FIG. 3 illustrates a plurality of ground predicates associated with auser u 301. To construct the user-centric graph, the approach firstconstructs a ground-predicate graph for each user. This is done usingall the ground predicates associated with that user. Theground-predicate graph for u 302 shows all the 22 ground predicates inthe example as a graph, where some predicates are represented as edgelabels for convenience. This ground-predicate graph is then representedby a single vertex in the user-centric graph 303.

When defining an edge, two vertices are used if certain conditions aremet. For example, u 303 and u′ 304 in the user-centric graph isdemonstrative. An edge may be defined between them 305 if and only ifany of the following conditions is satisfied: (1) a tweet of u mentionsu′ 306; (2) a tweet of u′ mentions u 307; (3) u is a friend of u′ 308;(4) u′ is a friend of u 309; (5) u is followed by u′ 310; (6) u′ isfollowed by u 311; (7) a tweet of u was retweeted by u′ 312; or (8) atweet of u′ was retweeted by u 313.

FIG. 4 illustrates the initial steps taken by the approach to processthe ground predicates and group them by user ID. This facilitates thegeneration of the user-centric graph and uses data parallelism via theMapReduce paradigm. The MapReduce paradigm is disclosed in J. Dean andS. Ghemawat, MapReduce: Simplified Data Processing on Large Clusters;Proc. of the 6th Symposium on Operating Systems Design andImplementation (OSDI), pages 1-10, San Francisco, Calif., December 2004,the contents of which are hereby incorporated by reference in theirentirety. The MapReduce paradigm refers to a parallel approach ofsplitting the input data into independent chunks, then applying mapoperations on the chunks in parallel, followed by grouping ofintermediate key-value pairs (generated by the map operations) by key,and finally applying reduce operations in parallel on each group ofkey-value pairs with the same key. As illustrated, an associative arraytweetUserCollection is generated by processing the predicates of thetype tweeted(.) 401. The array comprises data (key, value) pairs. In oneexample, key is the ID of a tweet and value is the ID of the user whoposted that tweet. This array is broadcast to all servers 1-m in thecluster at 402.

The input data contains ground predicates generated from tweets whereone ground predicate is stored one per line in HDFS at 403. The inputdata is split into blocks each containing b lines, with the exception ofthe last block containing less than b lines 404. On each block, the mapoperation is performed by invoking the subroutine MapPredicates 405,which can operate in parallel. The output data from all the mapoperations are shuffled and sorted as in MapReduce. The reduceoperations are performed in parallel by invoking the subroutineReducePredicates 406. The output data from the reduce operations arestored in HDFS 407.

FIGS. 5A and 5B illustrate the steps performed by a subroutineMapPredicates 500, which was invoked and performed at step 405 in FIG.4. First, a determination is made as to whether all of the groundpredicates are processed 501. If true, return back from the subroutine.Otherwise, the following steps are taken: Let p denote the next groundpredicate to process 502. Check if p is of type tweeted(.), verified(.),attacker(.), friend(.), friendsCount(.), followersCount(.),statusesCount(.), or favouritesCount(.) 503. Let u denote a userID inthe ground predicate p. If true, then output a key-value pair with u asthe key 504. Otherwise, check if p is of type malicious(.) ortrending(.) 505. If true, then output a key-value pair with p as the keyand the string “EXISTS” as the value 506. This is done to indicate thatmalicious(.) (in the illustrated example, a malicious link) ortrending(.) (in the illustrated example, a trending hashtag) exists andshould be treated specially later. Otherwise, check if p is of typecontainsLink(.) 507. If true, then output a key-value pair with a key oftype malicious(.) 508, thereby indicating a malicious link. Otherwise,check if p is of type containsHashtag(.) 509. If true, then output akey-value pair with a key of type trending(.), thereby indicating atrending hashtag at 510. Otherwise, check if p is of type mentions(.),isPossiblySensitive(.), or retweetCount(.) 511. If true, find the userIDu for the tweetID t in tweetUserCollection 512. Then, output a key-valuepair, indicating a user u and a predicate p, with u as the key at 513.Otherwise, check if p is of type retweeted(.) 514. If true, find theuserID u for the tweetID tin tweetUserCollection 515 and check if found516. If so, output a key-value pair with u as the key and a predicate oftype retweetedBy(.) as the value 517. (Note that retweetedBy(.) reversesthe order of variables in retweeted(.).) A warning is output if p is notrecognized 518. When all of the ground predicates have been processed bythe subroutine 501, it returns.

FIG. 6 illustrates the steps performed by the subroutineReducePredicates 600, which was invoked and performed at step 406 inFIG. 4. The subroutine checks if the first argument K_(i) is a userID601. If true, the arguments (K_(i),V_(i)) are output the way they are602. Otherwise, the second argument V_(i), which is a list of items, istested for the following condition 603: Is there any item in V_(i) thatmatches “EXISTS”? If false, then the subroutine returns. Otherwise,V_(i) is processed further. The subroutine proceeds to check if allitems in V_(i) have been processed at 604. If false, let t′ denote thenext item to process in V_(i) 605. If t′ is not equal to “EXISTS” 606,then it is a tweetID, and the corresponding userID ū is obtained fromtweetUserCollection 607. A pair (ū, Ki) is output 608. If all items inV_(i) have been processed 604, the subroutine returns.

FIG. 7 illustrates the next steps to construct the user-centric graph.The input HDFS file 701 is a set of (key, value) pairs, where the keydenotes a userID and the value is a single ground predicate (e.g.,malicious(http://a.com), trending(#sports)) or a list of groundpredicates associated with the userID. By applying map and reduceoperations 702, the input data is regrouped by userID to produce theoutput data 703 in the form of (key, value) pairs, where the key is auserID and value is the list of all ground predicates associated withthe userID. Next, the generated data 703 is split by key (i.e., userID)into individual (key, value) pairs 704, and each (key, value) pair isoperated by a map operation (i.e., MapOutWts) in parallel 705. Theoutput from all the map operations are shuffled and sorted as inMapReduce. Reduce operations (i.e., ReduceSumWts) are executed inparallel 706 to produce the user-centric graph, which is stored in HDFS707.

FIG. 8 illustrates the steps performed by the map operation denoted bythe subroutine MapOutWts 800, which was invoked and performed at step705 in FIG. 7. The key, which is a userID and the first argument of thesubroutine, is stored in variable u 801. The number of ground predicatesexcluding predicates of the type retweetedBy(.) in the value, which isthe second argument of the subroutine, is computed and stored in thevariable c 802. The pair (u, c) is output 803 and denotes a weightedvertex of the user-centric graph. The next task is to output weightededges associated with u. The variable edgeList is initialized to Ø 804.Next, a determination as to whether all ground predicates in the valuehave been processed 805 is made. If false, the next ground predicate isprocessed and if it is of type friend(.), isFollowedBy(.), mentions(.),or retweetedBy(.) 806, then an edge (x, y) is created between the twovertices where one of them is u 807.

The presence of the aforementioned predicates is an indication of socialrelationship between u and another user. Determine if edge (x, y) isalready present in edgeList 808. If true, then its weight w_(x,y) isincremented by 1 810. Otherwise, (x, y) is first added to edgeList andw_(x,y) is initialized to 0 809. After that w_(x,y) is incremented by 1810. When all of the ground predicates in the value have been processed805, edgeList is further processed to generate weighted edges. Next,determine if there are more edges in edgeList to process 811. If true,then output the edge (x, y) and its weight w_(x,y) 812. Otherwise,return from the subroutine.

FIG. 9 illustrates the invention's next course of action after all ofthe map operations are completed, to commence performance of thesubroutine ReduceSumWts at item 900, which was invoked and performed atstep 706 in FIG. 7. This involves shuffling the output data and sortingit as in MapReduce. An edge (x, y) may be repeated more than once in thesorted data. In such a case, all the weights associated with (x, y) mustbe added to produce a single weighted edge. This is accomplished byinvoking the reduce operation denoted by the subroutine ReduceSumWts900. Next, the system determines whether the first argument of thesubroutine is a userID 901. If true, the system outputs the arguments tothe subroutine as they are 902. Otherwise, the system adds up all of theweights in the second argument, which is the total weight of the edge903. Then the system outputs the first argument and total weight as theweighted edge 904. Once all the reduce operations have completed, theuser-centric graph is stored in HDFS, illustrated as item 102 in FIG. 1.

FIG. 10 illustrates the continued processing of the user-centric graphafter construction. Here, the user-centric graph is read as input 1001.This user-centric graph needs to be partitioned using graph partitioningmethods, which have been well-studied in the literature. Severalparallel methods are available for partitioning a graph into some numberof balanced partitions while minimizing the total weight of the cutedges (e.g., ParMETIS, ParHIP). Given m servers 1002, this graph ispartitioned (using an existing method) to create m balanced partitions(according to vertex weights) by minimizing the total weight of the cutedges 1003. Exemplary methods for partitioning are disclosed in K.Schloegel, G. Karypis, and V. Kumar, Parallel Static and DynamicMulti-Constraint Graph Partitioning, Concurrency and Computation:Practice and Experience, 14(3):219-240, 2002 the contents of which arehereby incorporated by reference in their entirety; and H. Meyerhenke,P. Sanders, and C. Schulz, Parallel Graph Partitioning for ComplexNetworks, IEEE Transactions on Parallel and Distributed Systems (TPDS),Jan. 26, 2015, the contents of which are hereby incorporated byreference in their entirety. By balanced, we mean the total vertexweight of each partition is nearly equal across all partitions. Thevariable i is initialized to 1 at step 1004. The system determineswhether the condition i>m at 1005. If false, the system collects all ofthe ground predicates in the ground-predicate graph of each vertex inpartition P_(i) at 1006. Then, the system stores these predicates into afile F_(i) on server i 1007. Increment i by 1 1008. If the condition i>m1005 is true, then the system proceeds to process the individual filesF₁ through F_(m). For every server j, the system runs an existingMLN-structure-learning method on F_(j) on server j at 1009. One exampleof a suitable MLN structure learning method for use at step 1009 isdisclosed in S. Kok and P. M. Domingos, Learning Markov Logic NetworksUsing Structural Motifs, Proceedings of the 27th InternationalConference on Machine Learning (ICML), pages 551-558, Haifa, Israel,2010, the contents of which are hereby incorporated by reference intheir entirety. This method identifies structural motifs in the data andconstrains the rules to be within these motifs thereby speeding up rulelearning.

Step 1009 results in an output of automatically generated first orderlogic rules that serve to predict malicious content and suspicious userson Twitter® or other social media. For example, FIG. 14 illustrates aset of first-order formulas 1401, 1402 resulting from step 1009.

All of the rules learned on all servers are combined by applying theunion operation to produce Rat 1010, which is provided to all servers1011. On each server j, run MLN weight learning on R using Fj 1012.

Step 1012 results in an output of automatically generated weights forthe first-order logic rules output in step 1009. For example, FIG. 15illustrates a set of output data detailing exemplary weights 1501, 1502for first-order formulas resulting from step 1009. The learned weightsreflect how likely the formulas are to be satisfied/unsatisfied in a setof possible worlds. Accordingly, with a weight of 3.5, formula 1501 islikely to be satisfied. Conversely, with a weight of −2.5, formula 1502is not likely to be satisfied, albeit with less certainty than formula1501 is predicted to be satisfied (i.e., the absolute value of theweight for formula 1501 is greater than that determined for formula1502).

Finally, the subroutine RuleRanking 1013 is used to select the mostrelevant rules in the KB.

FIG. 11 illustrates the steps performed by the subroutine RuleRanking1100 used to combine learned rules' weights on differentservers/partitions and illustrates the ranking of these rules based onrelevance. The first step is to compute for each rule, the product ofthe weights of the rule between every pair of partitions/servers. Thenthese partial products are summed to get a total weight. As the weightof a rule on a partition can be positive, negative, or zero, thedifferent weights are objectively combined. Because the weights of arule on two partitions are multiplied, when the signs differ for theweights, i.e., a rule is more likely to be satisfied in onepartition/server and is more likely to be unsatisfied in another, themethod penalizes such a rule. The total weight is used to rank the rulessuch that higher total weight equates to more relevancy of the rule.

The input to the subroutine is a set of n rules {r₁, . . . r_(n)}, andeach rule r_(i) has weights learned on m servers {w_(i1), . . . ,w_(im)} 1101. Starting with the first rule where i equals 1 1102, foreach rule r_(i), the method initializes the total weight W_(i) for ruler_(i) to zero and j to 1 1103. The variable k is initialized to 1 1104.A determination is made as to whether the condition j≠k holds 1105. Iftrue, then add w_(ij)×w_(ik) to W_(i) 1106. Increment k by 1 1107.Determine whether k is greater than m 1108. If false, go back tochecking the condition j≠k 1105. Otherwise, increment j by 1 1109.Determine if j is greater than m 1110. If false, reinitialize k 1104.Otherwise, consider the next rule by incrementing i by 1 1111. Check ifthere are any more rules to process 1112. If there are still rules toprocess, then go back to reinitializing W_(i) and j 1103. If all ruleshave been processed, then sort the rules in descending order by totalweight W_(i) 1113 and output the first K rules 1114.

The present invention may be carried out in a single or a plurality ofcomputerized systems working in tandem. An illustrative example is shownin FIG. 12 and generally indicated as 1200. Computerized system 1200that is shown in FIG. 12 may be considered to represent any type ofcomputer, computer system, computing system, server, disk array, orprogrammable device such as multi-user computers, single-user computers,handheld devices, networked devices, or embedded devices, etc. Forexample, and without limitation, computerized system 1200 is illustratedin FIG. 12 as comprising server, indicated as 1208, and a personalcomputer, indicated generally as 1204, which may be connected, asindicated by 1212, through a network, indicated generally 1216, to forma networked computer system using one or more networks (e.g., in acluster or other distributed computing system through a networkinterface).

For example, and without limitation, computerized system 1200 isrepresented schematically in FIG. 13 and is generally indicated as 1300that has a single computer, indicated as 1304, although it should beappreciated that computerized system 1300 may also include multiplesuitable programmable electronic devices. Computer 1304 typically mayinclude at least one processing unit (illustrated as “CPU” 1308) coupledto Memory 1312 along with several different types of peripheral devices(e.g., a Mass Storage Device 1316 with one or more Databases 1320 and1324, an input/output interface 1340, 1344, and a Network Interface(I/F) 1328. Memory 1312 may include (not shown) dynamic random accessmemory (“DRAM”), static random access memory (“SRAM”), non-volatilerandom access memory (“NVRAM”), persistent memory, flash memory, atleast one hard disk drive, and/or another digital storage medium. Massstorage device 1316 is typically at least one hard disk drive and may belocated externally to Computer 1304, such as in a separate enclosure orin one or more networked computers, one or more networked storagedevices (including, for example, a tape or optical drive), and/or one ormore other networked devices (including, for example, a server).

CPU 1308 may be, in various embodiments, a single-thread,multi-threaded, multi-core, and/or multi-element processing unit (notshown) as is known in the art. In alternative embodiments, Computer 1304may include a plurality of processing units that may includesingle-thread processing units, multi-threaded processing units,multi-core processing units, multi-element processing units, and/orcombinations thereof as is known in the art. Similarly, Memory 1312 mayinclude one or more levels of data, instruction, and/or combinationcaches, with caches serving the individual processing unit or multipleprocessing units (not shown) as is well known in the art.

Memory 1312 of Computer 1304 may include one or more applications(indicated schematically as “APP” 1332), or other software program,which are configured to execute in combination with the Operating System(indicated schematically as “OS” 1336) and automatically perform tasksnecessary for processing and analyzing sequences with or withoutaccessing further information or data from the Database(s) 6 of the massstorage device.

A user may interact with Computer 1304 via a User Input Device 1340(such as a keyboard or mouse) and a Display 1344 (such as a digitaldisplay) by way of a User Interface 1348.

Those skilled in the art will recognize that the computerized system1200 and computer system 1300 illustrated in FIGS. 12 and 13,respectively, are exemplary and may be in other configurations, mayinclude other components. Indeed, those skilled in the art willrecognize that other alternative hardware and/or software environmentsmay be used in computerized system 1200 and computer system 1300.

Some of the illustrative aspects of the present invention may beadvantageous in solving the problems herein described and other problemsnot discussed which are discoverable by a skilled artisan.

While the above description contains much specificity, these should notbe construed as limitations on the scope of any embodiment, but asexemplifications of the presented embodiments thereof. Many otherramifications and variations are possible within the teachings of thevarious embodiments. While the invention has been described withreference to exemplary embodiments, it will be understood by thoseskilled in the art that various changes may be made and equivalents maybe substituted for elements thereof without departing from the scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation or material to the teachings of the inventionwithout departing from the essential scope thereof. Therefore, it isintended that the invention not be limited to the particular embodimentdisclosed as the best or only mode contemplated for carrying out thisinvention, but that the invention will include all embodiments fallingwithin the scope of the appended claims. Also, in the drawings and thedescription, there have been disclosed exemplary embodiments of theinvention and, although specific terms may have been employed, they areunless otherwise stated used in a generic and descriptive sense only andnot for purposes of limitation, the scope of the invention therefore notbeing so limited. Moreover, the use of the terms first, second, etc. donot denote any order or importance, but rather the terms first, second,etc. are used to distinguish one element from another. Furthermore, theuse of the terms a, an, etc. do not denote a limitation of quantity, butrather denote the presence of at least one of the referenced item.

Thus, the scope of the invention should be determined by the appendedclaims and their legal equivalents, and not by the examples given.

That which is claimed is:
 1. A cyberthreat detection system comprising adistributed file system and a commodity cluster configured in datacommunication via a network, wherein the commodity cluster is defined asa plurality m of servers, each including a computer processor and anon-transitory computer-readable storage medium comprising a pluralityof instructions which, when executed by the computer processor, performsthe method comprising: receiving a data array characterized by a key anda value in a set of pairs relating to social media posts and users;storing a plurality of predetermined ground predicates; constructing aground predicate graph for each user reflected in the array;constructing a user centric graph having one or more vertices and one ormore edges and wherein each vertex represents the ground predicate graphcorresponding to each user; partitioning the user centric graph intobalanced portions P_(i) corresponding to the number of servers andwherein the ground predicates of each vertex in partition P_(i) arestored as a file on a server associated with that partition P_(i);determining a plurality of learned rules, in parallel on each server, onthe files stored on each server; receiving a union of the plurality oflearned rules; determining a respective weight for each of the pluralityof learned rules of the union; and ranking the plurality of rules of theunion by the plurality of weights.
 2. The system according to claim 1wherein variables in the predetermined ground predicates comprise one ormore of the following types (i) “tweetID” to denote the ID of a tweet(ii) “userID” to denote the ID of a user (iii) “link” to denote a URL(iv) “hashtag” to denote a word prefixed by the ‘#’ symbol and (v)“count” to denote a non-negative integer.
 3. The system of claim 1,wherein the data array is split into separate data blocks and whereinconstructing the user centric graph comprises: performing a mapoperation in parallel on each block; shuffling and sorting the dataoutput from the map operations; performing a reduce operation inparallel for each block of data shuffled and sorted; and storing thedata output from the reduce operation in a file.
 4. The system of claim3, wherein the file is a Hadoop Distributed File System file.
 5. Thesystem of claim 3, wherein constructing the user centric graphcomprises: splitting the data into key-value pairs; performing a secondmap operation in parallel in order to output weights; shuffling andsorting the output weights from the second map operation; performing asecond reduce operation in parallel on the output weights to produce theuser centric graph; and storing the user centric graph in the file. 6.The system of claim 5, wherein the file is a Hadoop Distributed FileSystem file.
 7. The system of claim 5, wherein an edge is definedbetween a first and a second user in the user centric graph if any ofthe following conditions are satisfied: a tweet of the first usermentions the second user; a tweet of the second user mentions the firstuser; the first user is a friend of the second user; the second user isa friend of the first user; the first user is followed by the seconduser; the second user is followed by the first user; a tweet of thefirst user was retweeted by the second user; or a tweet of the seconduser was retweeted by the first user.
 8. The system according to claim 1wherein ranking the plurality of rules of the union comprises:determining a respective partial product of the weights for each of theplurality of learned rules of the union; and summing the plurality ofpartial products to define a total weight.
 9. A computer-implementedmethod of cyberthreat detection, the method comprising: receiving, usingeach of a plurality m of servers, a plurality of first-order predicates;receiving, using a distributed file system, a plurality of groundpredicates associated with the first-order predicates and eachcharacterized by a key and a value; receiving a data array characterizedby a key and a value in a set of pairs relating to social media postsand users; storing the plurality of ground predicates; constructing aground predicate graph for each user reflected in the array;constructing a user centric graph having one or more vertices and one ormore edges and wherein each vertex represents the ground predicate graphcorresponding to each user; partitioning the user centric graph intobalanced portions P_(i) corresponding to the number of servers andwherein the ground predicates of each vertex in partition P_(i) arestored as a file on a server associated with that partition P_(i);determining a plurality of learned rules, in parallel on each server, onthe files stored on each server; receiving a union of the plurality oflearned rules; determining a respective weight for each of the pluralityof learned rules of the union; and ranking the plurality of rules of theunion by the plurality of weights.
 10. The method according to claim 9wherein variables in the ground predicates comprise one or more of thefollowing types (i) “tweetID” to denote the ID of a tweet (ii) “userID”to denote the ID of a user (iii) “link” to denote a URL (iv) “hashtag”to denote a word prefixed by the ‘#’ symbol and (v) “count” to denote anon-negative integer.
 11. The method of claim 9, wherein the data arrayis split into separate data blocks and wherein constructing the usercentric graph comprises: performing a map operation in parallel on eachblock; shuffling and sorting the data output from the map operations;performing a reduce operation in parallel for each block of datashuffled and sorted; and storing the data output from the reduceoperation in a file.
 12. The method of claim 11, wherein the file is aHadoop Distributed File System file.
 13. The method of claim 11, whereinconstructing the user centric graph comprises: splitting the data intokey-value pairs; performing a second map operation in parallel in orderto output weights; shuffling and sorting the output weights from thesecond map operation; performing a second reduce operation in parallelon the output weights to produce the user centric graph; and storing theuser centric graph in the file.
 14. The method of claim 13, wherein thefile is a Hadoop Distributed File System file.
 15. The method of claim9, wherein an edge is defined between a first and a second user in theuser centric graph if any of the following conditions are satisfied: atweet of the first user mentions the second user; a tweet of the seconduser mentions the first user; the first user is a friend of the seconduser; the second user is a friend of the first user; the first user isfollowed by the second user; the second user is followed by the firstuser; a tweet of the first user was retweeted by the second user; or atweet of the second user was retweeted by the first user.
 16. The methodaccording to claim 9 wherein ranking the plurality of rules of the unioncomprises: determining a respective partial product of the weights foreach of the plurality of learned rules of the union; and summing theplurality of partial products to define a total weight.
 17. A system fordetecting suspicious activity on social media comprising: a distributedfile system and a commodity cluster configured in data communication viaa network, wherein the commodity cluster is defined as a plurality m ofservers, each server including a computer processor and a non-transitorycomputer-readable storage medium; a data array characterized by a keyand a value in a set of pairs relating to social media posts and users;wherein the processor is configured to store a plurality ofpredetermined ground predicates; wherein the processor is configured toconstruct a ground predicate graph for each user reflected in the array;wherein the processor is configured to construct a user centric graphhaving one or more vertices and one or more edges and wherein eachvertex represents the ground predicate graph corresponding to each user;wherein the processor is configured to partition the user centric graphinto balanced portions P_(i) corresponding to the number of servers andstore the ground predicates of each vertex in partition P_(i) as a fileon a server associated with that partition P_(i); wherein the processoris configured to determine a plurality of learned rules, in parallel oneach server, on the files stored on each server; wherein the process isconfigured to perform a union operation to generate a union of theplurality of learned rules; and wherein the processor is configured todetermine a respective weight for each of the plurality of learned rulesof the union; and wherein the processor is configured to rank theplurality of rules of the union by the plurality of weights.
 18. Thesystem according to claim 17 wherein the processor is configured todetermine the plurality of learned rules by running a Markov logicnetwork learning algorithm.
 19. The system of claim 17, wherein an edgeis defined between a first and a second user in the user centric graphif any of the following conditions are satisfied: a tweet of the firstuser mentions the second user; a tweet of the second user mentions thefirst user; the first user is a friend of the second user; the seconduser is a friend of the first user; the first user is followed by thesecond user; the second user is followed by the first user; a tweet ofthe first user was retweeted by the second user; or a tweet of thesecond user was retweeted by the first user.
 20. The system according toclaim 17 wherein the processor is configured to ranking the plurality ofrules of the union by determining a respective partial product of theweights for each of the plurality of learned rules of the union andsumming the plurality of partial products to define a total weight.